Repository navigation
Mega MoE: optional per-token weights for fused shared experts - #56
Draft
gau-nernst wants to merge 1 commit into
Draft
gau-nernst wants to merge 1 commit into
gau-nernst wants to merge 1 commit into
Conversation
gau-nernst
force-pushed
the
megamoe-shared-expert-weights
branch
from
November 10, 2026 00:24
3b29b39 to
7d2eaee
Compare
Add `shared_expert_weights`, an optional FP32 `(num_tokens,)` tensor, to `fp8_fp4_mega_moe` (SM100, SwiGLU) and `bf16_mega_moe` (including the TRT-LLM weight layout). For shared-expert rows, the L1 epilogue multiplies the activation output by the token's weight, at the same place routed rows apply their top-k weight. This lets models that gate the shared expert per token, e.g. Qwen MoE's `sigmoid(shared_expert_gate(x)) * shared_expert(x)`, fuse it. The weights path is a template flag (`kWithSharedExpertWeights`), so `None` instantiates the same kernel as before: the only SASS difference is the kernel-parameter offsets shifted by the new pointer argument. The SM90 and SiTU kernels reject the weights; NVFP4 and FP4xFP4 Mega MoE do not take them yet. `tests/test_mega_moe.py` gains `--shared-expert-weights`, sweep cases with the weights, and covers them in the reference, the legacy baseline and the CUDA graph check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gau-nernst
force-pushed
the
megamoe-shared-expert-weights
branch
from
November 10, 2026 02:01
7d2eaee to
ee9be6f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
shared_expert_weights, an optional per-token weight on the fused shared expert, tofp8_fp4_mega_moe(SM100, SwiGLU) andbf16_mega_moe.Qwen MoE models gate the shared expert per token:
Mega MoE can fuse a shared expert today, but only as the unweighted
+ S(x), so these models have to run the shared expert separately. vllm-project/vllm#66659 uses this PR to fuse it.Design
shared_expert_weights: Optional[torch.Tensor] = None, FP32,(num_tokens,), contiguous, on the device ofy. The caller computes the weight, e.g.sigmoid(x @ w_s), so the kernel stays agnostic to the gating function.Nonekeeps the current behavior. It is the last argument everywhere, including the op schemas, so existing callers are unaffected. The checks are inline incsrc/apis/mega_moe.hpp, next to the other optional inputs.g * W2(h) = W2(g * h), the math is the same as scaling the output.kWithSharedExpertWeights(likekWithPostExpertNorm), soNonecompiles the same kernel as before. Astatic_assertrequires fused shared experts when the flag is set.y.Numerics: the kernel quantizes
g * hinto the L2 input, notg * quant(h), the same treatment routed experts get from their top-k weights.Tests
tests/test_mega_moe.pygains--shared-expert-weights:--emu-reference) and the legacy baseline weight the shared expert by it.--sweepgains six cases with weights. They cover zero and one token, multiple shared experts, thehot/empty/maskedroutings, and up to 8192 tokens. These cases are skipped on SM90.Results on an 8x B300 node,
--emu-reference, for the six sweep shapes with weights, run directly at 1 and 8 ranks withfp8xfp4andfp8xfp8, plus a Qwen-shaped case (hidden 2560, 512 experts, intermediate 640, top-10, 4096 tokens):max_token_diff≤ 2.8e-5, and 0 for the 0- and 1-token cases.devwithout this PR and without weights: the first--sweepcase at 8 ranks fails it, and so do the 1-token and Qwen-shaped cases with no weights. It is not caused by this change.--sweepstops at the first failure, and it fails at the same point ondevand on this branch with identical numbers:(8192, 7168, 128, 65, 8, 2, 10, 'masked')withmax_token_diff=1.79e-4 > 1e-4.Also tested:
bf16xbf16andfp8xfp4at 1 and 8 ranks. It checks power-of-two weights exactly with only the shared expert active, a sigmoid-gated linearity check with routing, every BLOCK_M the heuristic picks, an empty rank, and rejection of SiTU and misshaped weights.tests/test_bf16_mega_moe_trtllm.pypasses.WG_BLOCK_Mguard as the routed reload), the weighted cases at 1 rank (fp8xfp8,fp8xfp4) and the standalone test at 1 and 4 ranks (bf16xbf16,fp8xfp8,fp8xfp4) pass again.Performance
Fused vs. unfused gated shared expert
Without this PR, a gated shared expert cannot be fused. It runs as a separate BF16 MLP next to a MegaMoE that runs only the routed experts, as in vllm-project/vllm#64315. The comparison uses a Qwen MoE shape (hidden 2560, 512 experts with intermediate 640, top-10, one shared expert of 640), MXFP8 (
fp8xfp8), EP4 on 4x B300, as in the vLLM DP4 + EP4 serve. Times are in µs.tests/test_mega_moe.pyper-rank kernel time, range over two runs. The unfused arm isdevwith--num-shared-experts 0. The fused arm is this PR with--num-shared-experts 1 --shared-expert-weights.gate_upGEMM, fused SiLU·mul, BF16downGEMM, and a fusedy += sigmoid(g) * s. Measured on one GPU with a CUDA graph.Fusing the gated shared expert adds 0 - 3.5 µs to MegaMoE at decode sizes and about 9 µs at 4096 tokens per rank. It removes a 10 - 44 µs MLP. This is an upper bound: in vLLM the unfused MLP runs on a side stream and partly overlaps the router. vllm-project/vllm#66659 measures the end-to-end effect.
No cost without weights
tests/test_mega_moe.py, 8x B300, EP8, with one fused (unweighted) shared expert. Mean kernel time in µs, range over two runs:devNoneFor the
Nonepath, I compared the SASS of six kernel instantiations withdev. The opcode sequences are identical. The only difference is kernel-parameter offsets, which shift by 8 bytes because of the new pointer argument. The weights code is behindif constexpr, so theNonekernel does not change.Not done yet (draft)
🤖 Generated with Claude Code